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Laboratory Services Branch 
Quality Management Office 
125 Resources Road 
Rexdale, Ontario 
M9W-5L1 
(416) 235-6005 

April 1, 1992 

Dear MISA Workshop Participant, 

Many of the participants of the MISA Analytical Workshops held 
March 10, 11, 17 and 18th have expressed interest in a hard copy of 
the information presented. 

We have produced a proceedings/presentations document for each 
presentation package order form that the Quality Assurance Office 
received. Please share this with your colleagues who may have also 
attended, or other interested parties. As noted in the document 
introduction, please feel free to copy and use any portions you 
feel would be useful to your staff. 



Cammy Lynn Rape so 

MISA Analytical Workshop 
Co-ordinator 
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Th« 1992 MISA Analytical Workshops, hosted by the Environment 
Ontario, Laboratory Services Branch held March 10 & 11, 17 & 
18th, provided the opportunity for laboratory technicians, 
scientists and managers to present and exchange information 
related to the analytical support of Ontario' s 
Municipal/Industrial Strategy for Abatement (MISA) Program. 

Over the two 2-day workshops, through formal presentations, 
small workgroups and open discussion, a lot of information and 
ideas were tabled. 

As a direct result of participation and comments from you and 
your colleagues. Ministry Laboratory staff were able to re- 
design the second workshop (March 17, 18) to be more 
effective. This is identified in the assessment survey 

summary which follows . 

As a follow-up the the workshop, and as a means of capturing 
most of the information presented, this 
proceedings/presentations document was compiled. 

These proceedings provide hard copies of information 
presented, and in most cases, in the exact format (slides, 
overheads) used by the authors. This was done intentionally 
to promote further use and discussion by workshop attendees . 
Please feel free to copy and use any portions you feel would 
be beneficial to your colleagues and staff. 

On behalf of Environment Ontario Laboratory Services Branch, 
Management and staff, thank you for attendence and 
participation . 



G . Crawford 

Manager, Trace Organics Section & Quality Assurance 

Laboratory Services Branch 

April 2, 1992 
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SPEAKER ABSTRACTS 
DAY ONE (SESSION ONE) 

A) MISA OVERVIEW 

Patricia Baulu, MOE/Lab Services Branch 
T«l: (416) 323-2770 
Fax: (416) 323-2785 

A brief historical overview of the MISA Program is given. The 
status of the MISA program is discussed as well its future 
direction . 

B) CODE OF PRACTICE DOCPMENT 

George Crawford, MOE/Lab Services Branch 
Tel: (416) 235-5757 
Fax: (416) 235-6110 

Several Code of Practice documents exist here in Canada and 
internationally in Europe and should be considered when 
setting up a laboratory. These documents are briefly reviewed 
and contacts are provided. 

C) NEW DEVELOPMENTS WITH NIMMP 

Peter Campbell, MOE/Lab Services Branch 
Tel: (416) 235-5872 
Fax: (416) 235-6107 

A general overview of NIMMP is given. New improvements in the 
approach taken to carry a procedure through the NIMMP process 
are discussed. 

D) BULLETIN BOARD 

Cammy Lynn Raposo & David Boomer, MOE/Lab Services Branch 
Tel: (416) 235-6005/5858 
Fax: (416) 235-6110/6113 

Electronic communication has now become a fast and efficient 
way of "sharing information." The LSB QM Bulletin Board is 
introduced by way of discussion and a visual demonstration. 
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B) lAB PC PRACTICES 

David Boomer, MOE/Lab Services Branch 
Tel: (416) 235-5858 
Fax: (416) 235-6113 

Advances in computerized instrumentation has not only 
increased instrumentation sensitivity but has also improved on 
analyte selection and QC practices. Two of these systems in 
place at the LSB are QC-Expert and LQAS . 

DAY ONE (SESSION TWO) 

f) ROUND ROBINS - MOE lAB PERSPECTIVE 

Sylvia Cussion, MOE/Lab Services Branch 
Tel: (416) 235-5842 
Fax: (416) 235-6110 

The Ministry of the Environment Lab Services Branch has been 
co-ordinating Round Robins since 1988. This presentation will 
provide a brief overview of the program and some examples of 
past round robin data. 

6) ROUND ROBINS - COMMERCIAL LAB PERSPECTIVE 

Judith Szekely, Hamilton-Wentworth Regional Laboratory 
Tel: (416) 546-4454 
Pax: (416) 545-7177 

Brian Fowler, Axys Analytical Laboratory 
Tel: (604) 656-0881 
Fax: (604) 656-4511 

Round robin studies are conducted and particpated in by many 
laboratories. In different perspectives, presentations are 
provided by both a Regional laboratory and a commercial 
laboratory . 

H) ROOND ROBINS - WTC PERSPECTIVE 

Mohammad Foroutan & Peter Fowlie, Wastewater Technology Centre 
Tel: (416) 336-4855 
Fax: (416) 336-4765 

The Wastewater Technology Centre in Burlington has played a 
major role in conducting round robins in both water and 
sediment matrices. The WTC will provide an overview of how 
they conduct round robin studies and some past examples of 
round robin data. 



I) ANALYTICAL CHEMISTRY UNDER RSGUIATIONS / JAWG 

Dr. Otto Herrmann, Ontario Hydro 
Tel: (416) 231-2877 
Fax: (416) 231-9679 

When regulations are imposed on laboratories, analytical 
chemistry becomes more challenging. These challenges are 
discussed in a industry/commercial/government laboratory group 
called JAWG (Joint Analytical Working Group) . 

DAY TWO (SESSION THREE) 



J) PROTOCOL DOCTJMENT 

Patricia Baulu, MOE/Lab Services Branch 
Tel: (416) 323-2770 
Fax: (416) 323-2785 

The Protocol for the Sampling/Analysis of Municipal/Industrial 
Wastewater is introduced. Parts of the document are displayed 
in the presentation to familiarize the participants with its 
features , 

K) POTENTIAL NON-COMPLIANCES /ENFORCEMENT 

Cathy Doehler, MOE/Lab Services Branch 
Tel: (416) 235-6055 
Fax: (416) 235-6110 

Inspections of industry and laboratories are required under 
the MISA regulation. This presentation focuses on the LSB and 
their role in inspections and enforcement of the MISA 
regulation. 

L) DETECTION AND LOW-LEVEL COMPLIANCE /ENFORCEMENT 

Don King, MOE/Lab Services Branch 
Tel: (416) 235-5838 
Fax: (416) 235-6110 

With the completion of the MISA Monitoring phase, the MISA 
quality control data is being submitted to the LSB for review. 
This presentation provides a comprehensive overview of the 

fundamental concepts of quality control and the interpretation 
of low-level data. 



M) CAEAL UPDATE 

Bill Traversy, CAEAL/ 
Tel: (613) 562-2200 
Fax: (613) 562-2203 

George Crawford, MOE/Lab Services Branch 
Tel: (416) 235-5757 
Fax: (416) 235-6110 

The Canadian Association for Environmental Analytical 
Laboratories (CAEAL) is introduced. Information is provided 
about membership, certification and accreditation. 



SUMMARY OF PARTICIPANTS ASSESSMENT OF 
THE MISA WORKSHOPS 1992 

At Session 1 (March 10, 11), there were 73 participants with 35 
returning their questionnaires. At Session 2 (March 17, 18), there 
were 50 participants with only 10 returning questionnaires. Based 
on those questionnaires received, graphical representations of the 
siimmarized results are shown below. 
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SECTION TWO 
DAY 1 - SESSION ONE PRESENTATIONS 



CHAIRPERSONS - B. BERG, MOE 

C. DOEHLER, MOE 



MISA OVERVIEW - P. BAULD, MOE 

CODE OF PRACTICE DOCmffiNT - G. CRAWFORD, MOE 

NEW DEVELOPMENTS WITH NIMMP - P. CAMPBELL, MOE 

BULLETIN BOARD - C. RAPOSO & D . BOOMER, MOE 

lAB QC PRACTICES - D. BOCMER, MOE 
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MISA OVERVIEW 

Background 

OFFICIALLY ANNOUNCED: JUNE 1986 WHITE PAPER 
PURPOSE: REGULATE DISCHARGES 

IJECTIVE: VIRTUAL ELIMINATION OF PERSISTENT 
TOXICS FROM INDUSTRIAL/MUNICIPAL 
DISCHARGES 
PROACH: DISCHARGERS MONITOR EFFLUENTS 
LIMITS SET BASED ON DATA & BATEA 
JTC'S: REGULATIONS DEVELOPED IN CONSULTATION 
WITH INDUSTRY, MAC, ENVIRONMENT CANADA. 



MiSA OVERVIEW 

Background 
INDUSTRIAL SECTORS: 

9 SECTORS COVERING 300 DIRECT DISCHARGERS 

Petroleum Refining 

Organic Chemical Manufacturing 

Iron & Steel 

Inorganic Chemical 

Pulp & Paper 

Metal Mining 

Metal Casting 

Industrial Minerals 

Electric Power Generation 



MISA OVERVIEW 

Background 



MUNICIPAL SECTOR: 



DIRECT DISCHARGERS: 400 STP's 

INDIRECT DISCHARGERS: 12 000 + 

22 sectors 






MISA OVERVIEW 

Current Status 
MONITORING: COMPLETED FOR ALL SECTORS 

DATA EVALUATION: 4 SECTORS COMPLETE 

5 SECTORS IN PROGRESS 

CLEAN-UP OF DATABASE FOR EACH PLANT: 

Correction of data entry errors 
verification of outliers 

QA/QC ASSESSMENT: 

validate data for use in setting limits 



MISA OVERVIEW 

Issues Resolution 



ISSUES RELATED TO LIMITS SETTING RESOLVED 
AND PUBLISHED BY MOE IN CONSULTATION 
WITH INDUSTRY 



PURPOSE: SET-UP STANDARDIZED SET OF 
PROCEDURES AND CRITERIA TO ENSURE 
CONSISTENCY ACROSS SECTORS 



MISA OVERVIEW 

Issues Resolution 

ISSUES : 

BAT and BATEA 

Selection of parameters 

Monitoring data evaluation 

QA/QC 

Limits setting and form of limits 

Monitoring for assessment 

Reporting to the public 

Virtual elimination 

Flow measurement accuracy 

Net loadings 

By-passes 

Stormwater 

Toxicity 

Compliance 



MISA OVERVIEW 

Limits setting 

SELECTION OF PARAMETERS: 

• compounds selected as candidates for limits 

• list derived from computer data base according 
principles set out in IRC report 

• only parameters validated in QA/QC assessment 
are selected for BAT evaluation 

• only parameters for which treatment exists (BAT) 
are targets for limits 

• draft limits are calculated using statistical treatment 
outlined in IRC report 

• draft limits are reviewed/approved within MOE 



MISA OVERVIEW 

Limits setting 

• "FINAL" LIMITS PRESENTED TO JTC FOR 
CONSULTATION 

• LIMITS REGULATION, DEVELOPMENT DOCUMENT & 
ASSOCIATED PROTOCOLS UNDERGO PUBLIC 
REVIEW 

• LIMITS REGULATION PROMULGATED 

CURRENT STATUS : 

• DRAFT LIMITS DEVELOPED FOR PETROLEUM & 
PULP & PAPER SECTORS 

• LIMITS UNDER REVIEW WITHIN MOE 



MISA OVERVIEW 

New Initiatives 

ZERO DISCHARGE : ELIMINATION OF PERSISTENT 
TOXICS FROM EFFLUENTS DISCHARGED TO THE 
ENVIRONMENT THROUGH BANS & PHASE-OUTS 

TOXICS REDUCTION : REDUCTION/SUBSTITUTION OF 
TOXICS USED AND/OR RELEASED 

POLLUTION PREVENTION : ELIMINATE IN-PLANT USE 
AND PRODUCTION OF TOXICS AS OPPOSED TO 
END-OF-PIPE TREATMENT 



MISA OVERVIEW 

Summary 

REGULATIONS : EXPECTED FOR PUBLIC REVIEW IN 
SPRING 1992 FOR PULP & PAPER AND PETROLEUM 

DATA REPORTS : PUBLISHED FOR IRON & STEEL AND 
ORGANIC SECTORS 

QA/QC REPORTS : UNDERGOING FINAL REVIEW IN 
IRON & STEEL, ORGANIC AND MINING SECTORS 
PRIOR TO FINAL PARAMETER SELECTION/LIMITS 

DATA BASE CLEAN UP : NEARING COMPLETION 

REGULATIONS: SCHEDULED FOR END OF 1992 



ONTARIO MINISTRY OF ENVIRONMENT 



LABORATORY SERVICES BRANCH 



MISA ANALYTICAL WORKSHOPS 



MARCH 10&11. 1992 



MARCH 17 & 18, 1992 



I 

T 




yALIIY 









Doing the 

right things 

right the first time, 

on time, 

aH the time — 

and always 

to the customer's 

satisfaction!'" 



The gufllitv Mannopment Tnstii-nt« -'a n^vic^-inn of r.R a 



Ml 



CODE OF PRACTICE/ DOCUMENT(S) 

ISE 



o 



Definition 



° Application 
° Content 



Q 



Sources 



CODE OF PRACTICE/ DOCUMENT(S) 

ISE 



Pefii iition 

A Collection of Rules or Canons, Systematically 
Arranged, that circumscribe the conduct, 
organization and operation of an analytical laboratory 
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/VPPLICATION 

o A COP is essential to establishing and demonstrating 
the viability of an analytical Laboratory 

o All Accreditation and Certification Programs demand 
an outline of COP information as a mandatory 
requirement (CAEAL, SCO, CSA, A2LA, lAETL) 

"^ Demonstrated adherence to a COP is one measure of 
a Laboratory's suitability to provide "MISA Service" 
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INTRODUCTION 



I 

;. ■ 

This Directory contains a list of accredited testing organization^. 
and the registered detailed scopes of their accredited testing 
capabilities. Some of these involve the calibration of testing a-' 
measuring equipment; such "calibration laboratories" are included i-| 
the generic term "testing organizations". Accredited testin" 
organizations meet the criteria of the Standards Council of Canadii 
aocuaent AccTsditation Criteria and Requiremenzs for TestiriM 
Organizations (CAN-P-4A) , 

I 



.The accreditation criteria deal with the adequacy of administration, 
human and physical resources, quality system procedures and freedo' 
rrom conflict of interest. The criteria are compatible with ISO/I"v 
Guide 25 General Requirements for the Technical Comoetence ofm 
lestzng Lanorat ories . ' ■ 

Accreditation is the formal recognition of the capability of a.l 
organization to manage and perform specific tests registered witl. 
the Council. It does not guarantee individual test results, nor ii 
it a guarantee that test results will be accepted by product 
certification bodies. 

Accreditation does not limit an organization to conducting only 
tnose tests for which it is accredited. Most accredited organiza-i 
-ions carry out other tests, A client, may request that aiP 
accredited testing organization indicate in its report whether or^ 
not a test is registered. ■ ■ 

The role of the Standards Council is • to recognize competent testin J 
organizations in Canada, through accreditation, and to orovide the" 
-— ^.ant j^niormation to potential clients. Business matters between^ 
an accredited testing organization and its clients are legal! 
--.-nsacticns strictly between the two narties. 



™cd"/t~£^''^?!L.il ^^^^^ -^ prcmota the services 

♦.»„^^„^ . '"~ - --- --------^. -- -:s£rs aiio. potenuiai users 

om!.;'/ "/. "■ ^^=° serves to publicize the program 
urganizations wn-ff»h r-ntti^ %,«««r,**. -c , . . . r o 
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Quality Products 



Attitude, Commitment + Service 



Quality Control 
Quality Assurance 
Quality Assurance Manual 
Quality Management Plan 
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Quality management and quality assurance standards 
— Guidelines for selection and use 
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DATAREPORTD^G 

10.1 Definitions 32 

10.2 Round-off 33 

10.3 Verification 33 

10.4 Data Quality 33 
10 J Dam Intcrpietation 34 

10.6 Low-level Data Reporting ■ 34 

10.6.1 Option a: Withhold 35 

10.6.2 Option b: Rcpon 36 

10.7 Remark Codes 36 
IL RECORDS AND DATA MANAGEMENT 

11.1 Operational Documentation 37 
11.1.1 Documentation Approval 37 

11.2 Data Management 37 
11.2.1 Analytical Records 37 
11J2.2 Quality Control Records 37 

11.2.3 Record Security 3g 

1 1.2.4 Recoid Corrections ' 38 
1 1J2.5 Data Bass Corrections 38 
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nOHPAT^TRON O F REQUIREMEm-S TN STANDARDS (*) 



^^ 



ISO Guide 25 General Requirements for Technical Competence of Testing Labs. 

EN 45001 General Criteria for the Operation of Testing Labs. 

ANSI/ASQC 92 Model for QA in Production and Installation. 

EPA 40 CFR Pt 792 TSCA GLP* 

EPA 40 CFR Pt 160 FIFRA GLP* 

FDA 21 CFR Pt 58 GLP* 

ANSI/ASQC Q2 Quality Mgt. & Quality Sys. Elements for 

Laboratories -Guide lines 

The latent of the comparisons is to show that the ANSI/ASQC Standard covers 
the majority of the requirements in the laboratory standards and adds some 
additional requirements, which nay have to be addressed by the laboratory 
that is doing manufacturing control, lot acceptance, or support work. The 
subject is addressed in the paragraphs indicated but it is not intended that 
the exact wording is the same or that the same level of detail is required. 
If a single section number addresses basically the sajne subject and all 
subheadings, it is listed only once. The initial breakdown was on Guide 25, 
therefore, the subjects are arranged in the order shown in Guide 25. 



If there is a topic which could not be located, there will be '-- 
that stfmdard heading. 



under 



Since ANSI/ASQC 92 addresses "Production" rather that test laboratory the 
following was used as a corollary for the leboretory: 
Product - Test results and reports 
Processes - The test methods and other procedures 
Calibration - Standardization 
Raw Materials - Reagents, etc. 
Quality Assurance system - The systen normally used to 

produce products but not generally found is laboratories 

as an organized separate activity. 
Service - Equals product. The laboratory service (product) is 

test results, technical competence and Assurance. 

* Require a detailed protocol and other biological requirements related to 
non-clinical studies but the requirements are being applied to testing 
laboratories that supply identity, purity, physical property, and 
specification testing for materials being subjected to the non-clinical 
studies. Only those paragraphs related to "testing" are referenced. 
Those related to non-clinical studies (animal, biological, etc. testing) 
are not included. However if "animals" were used, the sections of the 
GLPs which cover their handling, generally, have the same types of 
requirements as those listed under reagents, equipment, facilities, and 
records in the other standards. 
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DRAFT 



DP.AFT 



i -(*) Adapted from " Quality Assurance for the Analytical Laboratory ", 

Tne CeriTre tor Proressional Advancetnent , East i-runsv.-ick, New Jersev, l.-;a. 
Jack E. Weiier, Dow Corning Corp., Mich. 



Comparisons of Standards 
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Subject 



Scope and field of application 
Testing laboratories 

Specific location 

Specific methods 
Statement of Compliance 

Definitions 

Organization 

Legally identifiable 

Organization chart 

Quality systeia 

Competence 

Freedom 

Extent and limits of 

responsibility 
Technical management 
Proprietary and confidential 

security 
Current training 
Responsible management 



Comparison of Standards 
25±_ 



45001 



il22_ 



FDA/EPAJ? 



1-H- 



1.1/1.2 



2(5)^ 



2(13) 
3 

S.l 

5.1 

4 

5.1 

5.1/5.4 
5.4.6 

5.2 
5.1 



4.3 



S.4.2 



3CA3)*^ 

4.1.2 

1.2.1 



4.13 

4.12.] 

4.3 

4.4 

4.1.3 



I/.IO 

--/■12,', 

.3 
Sub B 



Quality System 4. 5.4.2 4.2 

Quality Policy -- 5.2 4.1.1 

One responsible individual 4.1 5.4.2 4.1.2.3 

Quality Manual 4.2 4.2 

Structure (chart) 4.1.2 

Operational and functional duties 4.1.2.1 

Quality Assurance procedures, general 
Quality Assurance procedures, for tests 
Proficiency testing, standards, etc. 
Corrective action procedures 
Complaint procedure, technical 
Contract review 
Change control (document) 
Management Quality System Review 4.3 

periodic & documented 
Quality Loop 

+ = The listing was established based on Guide 25 order 
++ = When a single number is shown at the top of a section, it 
generally covers as subsections and additional numbers 
indicate that the subject is addressed in other places. 
* ( ) = Number of definitions listed. 
** A = ANSI/ASQC A3 & IS08402 
# = Subsection reference only 
Vif = --/.12 means not in FDA GMP but in the EPA GMPs 



MOE 

1++ 
4++ 



4(11) 2(9)/8(8) 

9(7)/10(5' 





5.3 
5.2 
6.2 


4.2 
3++ 




5 


2++ 


31/. 33 


14.1.2 


3.4 


.31f 






-- 


14.1 


4.3 


--. 


6.2.1 


3 


.35 


5.2/6.0 


2 




5.1 


3 




5.3 




.81 


6.3/5.3 


3.4 
3.5.] 




6.2 


3.5.2 




6.2.5 


3.5.3 




9.4/16/17 


3.5.5 


.31g 


12.0 


8++ 




18.2 


3+* 


-- 


6.1 
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Syb .jgQt 



25 


45001 


092 




FDA/EPA/f 


2 


MOE 


5.1 


5. 


4.1.2. 


2/ 




.29 


14.0 


4.2 


5.1 




4.17 










4.3 


5.2 














4++ 


5.3 














4.2.3 


5.4 










.31b 




^ 


5.4 












14.4 


4.3.3 


-- 


-- 


mm 






.31 


-- 


4.2 






4.10 






.61 


9 


7 + + 


6.1 


5.3.1 






.31c 


& .61 




7.2 


6.2 


5.3.3 








.63b 




7.2.4 


6.3 


5.3.1 












7.2.3 



Staff 

Education, training, & 

technical knowledge 
Job descriptions for senior technical 
Adequate supervision 
Suitable staff to cover absence 
Records of qualifications, & training 
Ms2-seni?r.t 

Equipment 

Adequate equipment for scope 
Proper maintenance of equipment 
Control of unusable equipment 

(Tags) 

Use of outside equipment -- 5.3.1 

Records for each major item: 6.4/6.5/ 5.3.3 4.9.1 .63c 
Name of equipment 6.6 

Manufacturer's name, type 6e 

serial No. 
Date received and Date of -- 4.5.4 7.1 

initial service (Condition of 
receipt - new, used, etc.) 
Current location .#Se 

Details of maintenance 
Date of last calibration (reports) 
Neximum calibration interval 
Label or tag with calibration date 
and due date 

Calibration 5.3.3 4.9.2 

Before use and to established 7.1 

program 
Traceable to National Standards, 7.2 — "■■ 

international standards or 

correlation inter-lab studies 
Reference standards with single use 7.3 
Standards traceable to national or 7.4 

international standards 
Checks between calibration (where 7.5 

revealment) 
Reference materials traceable to 7.6 

reference standards 
Characterization of materials — -• 



9.0/9.6/ 
63 8.3 


8/9--J 
8.3 ■ 


105 


8 . 3 . fl 


.107 9.6.3 


8.3.% 
8.3.^ 




8.4 




8.3.-1 


M35 


8.3.:^ 



FDA/EPAfl 


2 MHF 


31e/.63b 


6-3-V8. 07970 + + 


.81 


6.3.5/ 8.1.1 




6.3.6 8.1.1 




11.1 



Comparisons of Standards 
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Subiect 25 45001 Q21 

Test Methods and Procedures: 5.4.1 4.8.1 

Adequate documented instructions 8.1 
Specifications available to staff 8.2 
Non-standard methods fully 8.3 

docuioented 
Checks of manual calculation and 6.4 

data transfer 
Control of electronic data fi.S 11 2 S 

processing 
Subcontracted work '•- 5.4.7 4.5.2 9.5/13.0 

Environment 5.3.2 -- .41 11.0 5.0+ + 

Controlled to net effect the test, 9.1 .43 5.1.2 

staff , etc. 
Controlled access by outside 9.2 

persons 
Good housekeeping 9.3 5 2 

Handling of items to be tested. 5.4.5 4.4 .47 8.1/8.2 6.0++ 

System to identify samples/test 10.1 4.8.1 5_4 

items 4.7 

Bonded storage, if necessary 10.2 6,3.1 

Protection from damage 10.3 

Receipt, retention and disposal 10.4 .95 6.3.7 

procedures 

Records 5.4.4 4.9.4 .33f 6.3.7/8.4 11++ 

Suitable for repetition of test- 11.1 & .51 11.1 

original observations, 4.15 .130 

calculations , calibration and .195 11.2 

final report 
Meet local law requirements 11.2 

Secure and confidential 11.3 1 i "■? 

Comparisons of Standards -- — — . 190 11.2.3 
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Subiect 



.15. 



45001 



12.1 



12.2 



Q92 



Test Reports 5.4.3 4.1.4 

Accurate, clear and unambiguous on 
presentation of results and 
revealment information 
Include the following: 

Name and address of laboratoiry 
Unique identification (serial No.) 

on all pa?e5 
Name and address of client 
Description and identity of test item 
Date of receipt of item and testing 
Test results relate to itecj tested 
Identification of method, procedure, 

specification, etc. 
Description of sampling procedure 
Deviations, etc. from method 
Disclosure of non-standard methods used. 
Support data - charts, etc. 
Statement of measurement uncertainty 
Signature and title of technically responsible 4.9.3 

individual and date of issue of report 
Statement about reproduction without approval 
Nc advice or recommendations 

Explanation of extrapolation from sample to lot 
Format of report controlled and 12.3 

standardized 
Control of correction (supplement 12.4 -- 4.12 

to report) 
Data Validation 



Client cooperation -- 6.1 

Accreditation Body cooperation -- 6.2 

Other lab cooperation (round robin) -- 6.3 

Accreditation responsibilities -- 7. 

Purchasing control 

Incoming acceptance materials 

Client supplied product 

Special processes 

Receiving inspection 

Test status 

Internal audits 

CTJp 

Heagent control 

Economics 
ih) FDA/EPA do not accredit. It addresses audit, response, etc 

aicressed this subjecu related to iegal actions. 
•-??# Effects of non-comoliance 



4.5 

4.5.5 

4.6 

4.8.2 

4.9.1 

4.11 

4.16 

4.18 



FDA/EPA,^.' 



.35b7/ 
.33a8/ 
.185 



MOE 

10 + + 

X 



X 
X 
X 
X 
X 



.10 

■ nam 



10.5(1) 

10+ + 

11.2.4 

10 10.3/10.4 

1.3 3.3 

3.6.1 

7.0 I 



.3Sbl 



.83 
.200 also 



18 
15 

1.4/19 



6 + + 
3.6 



U) Provision of Data Interoretation 



CODE OF PRACTICE/ DOCUMENT(S) 

ISE 



c 



Definition 



Application 
Content 



o 



Sources 



I 



STANDARDS OF INTEREST TO LABORATORIES 



* ISO Guide 25 General Requirements for the Technical 

Competence of Testing Laboratories 

ISO Guide 38 General Requirements for the Acceptance of 

Testing Laboratories 

* MOE Code of Practice for Environmental Labaoratories 

* CSAZ201-M1990 Qualification Code for Ontario Laboratories 

analysing Industrial Waste 

EN 45001 General Criteria for the Operation of Testing 

Laboratories 

* ANSI/ASQC 02 Quality Management & Quality Systems Elements 

for Laboratories - Guidelines 

Under Construction: 
MOE/CAEAL Practical Guide for Laboratory Analysis of 

Environmental Samples 

EPA (DRAFT) Good Automated Laboratory Practices 

Recommendations for Ensuring Data Integrity In 
Automated Laboratory Operations with 
Implementation Guidance 



<' .■0.on"^ 



-J 



Mailing List 



Documents Contact(s) 

CSA Canadian Standards Association 

ISO Consolidated Mailing List 

178 Rexdale Boulevard 

Rexdale, Ontario 

Ph (416) 747-4368 

or 

Quality Management Institute 

(A Division of CSA) 

Kississauga Executive Centre 

Suite 1420 

4 Robert Speck Pkwy 

Mississauga, Ontario 

L4Z ISl 

Ph (416) 272-3920 

FAX (416) 272-3942 

(Note: CSA and QMI are Canadian Distributors 
for ISO) 

MOE Ministry of Environment 

Laboratory Services Branch 

Quality Management Office 

P.O. Box 213 

Rexdale, Ontario 

M9W 5L1 

Ph (416) 235-5758 

FAX (416) 235-6110 

USEPA United States Environmental Protection Agency 

Office of Information Management 
Research Triangle Park 
North Carolina 
27111 

ANSI/ASQC American Society for Quality Control 

Customer Service 
P.O. Box 3066 
Milwaukee, Wisconsin 
53201 
Ph (800-248-1946 TOLL FREE) 



WHAT WAS 

NIMMP? 
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I 
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M 
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NEW 

INSTRUMENTAL 
MEASUREMENT 
METHOD 




M 
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PRINCIPLES & PROCEDURES 

"HOW, NOT WHAT OR WHY! 

hIMMP MAY EXPAND TO ENTIRE MISA REGS 

BUT ONLY TO IMPROVE THE "HOW" 

V^^HA'f AND WHY ARE ADDRESSED 
BY RSQULATION-SETTJNQ OFFICE 

PRINCIPLES • A FEW KEY WORDS 



PROCiEDUREg ^ DETAILED 8PEC»RGATI0NS 




WHAT WILL HAPPEN? 




iff 

dRiTBRfA/rrdfrrKxwi 



I 




PROPOSAL 

MUST SHOW IDEA: 




•^> 



'*> 



"WORKS" BEYOND^ REASONABLI 
DOUBT 

r/IECTS THE MDL (ANALYTrCAL) 
liENEFITS USiRS OR QOVT^ 



MOE WINDOW 




• ONE CONTACT TEL. NO. & ADDRESS 

• V\1NDbW SETS UP EXPERT PANEL 

• KNOWS STATUS OF PROPOSALS 

• CLOSES FILES \NHm PROPOSAL IS 
ACCEPTED OR REJECTED 

• P/XSSES APPROVED PROPOSAL TO 
REGULATION TEAM 



EiXPERT PANEL 



t ':XPt:RT(S) WILL REVIEW PROPOSAL 

'> PANEL" WILL SET CRITERIA & TIMEFRAME 

'■ PANEL- WILL WORK WiTH PROPOSER 

" ACCiiPT/REJECt PROf'OSAL WITH REASONS 

»• J^EPORT BACK TO WINDOW AND PROPOSER 




CHANGE REGULATION(S) 



o 



u 



o 



f) 



PRINCIPLES AND PROCEDURES 
aXIfJTINQI REGULATIONS -SLOW 
MEV/ REGULATIONS -WHEN THEY COME 
MCni STRIVES TO RETAIN FLEXIBILITY 




CHANGES TO NIMMP 

(UNDER DISCUSSION!) 




• EXPANDING AND GENERALIZING ROLE 

• ONE WINDOW -GUARANTEED RESPONSL^^^^ 

• fiXPERT PANEL (QUIDANGE DOCUMENT) ^^ 

• OEVELOP CftlTEHIA AND TIMETABLE ^^ 
WITH PROPOSER ^^ 

JJIMPLIFf^D EVALUAtfOKl PROCESS ^^ 



y 







E LECTRONIC COMMUNICATION 

What's New At The MOE Lab? ... 



E LECTRONIC COMMUNICATION 

Will affect how we work by ... 



H Allowing new types of task structures. 

■ Fostering new types of reporting relationshi[)s. 

■ Promoting a more democratic forum where ideas 
can be exchanged spontaneously and casually. 



E LECTRONIC COMMUNICATION 

An Electronic Bulletin Board Could Be Used For.... 



Messages 



B Exchanging Files 



Peer Review of Papers, Methods, Presentations 



H Asking "Does Anybody Know...? questions, 



Bullv^tins Announcing Upcoming Events. 



El -ECTRONIC COMMUNICATION 

What is Required? ... 



R Computer; any computer with a communication 
pori. 

n Moc^em; any modem. HAYES or HAYES-compatible 
are easiest to work with: 

2400 baud is maximum transmission rate at this 
time. 

H Software; Any software that will handle the 
modem, (e.g. PC-DIAL, PC-TOOLS, TELIX, 
WINDOWS ETC.) 



EL ,ECTRONIC COMMUNICATION 

MOE Quality Management BBS... 



■ The MOE QM BBS is currently being run in 
the QM Office. 



B It was developed to: 

- allow users to obtain MOE analytical methods 

- act as an "information exchange center" for 

Lai), MISA and QM issues between MOE and clients 

■ Will reduce paper usage, postage costs, time 
and money 



E LECTRONIC COMMUNICATION 

How Do We Access The BBS? ... 



Dial (416) 235-6136 from your computer to log 
on the MOE QM BBS. 

Follow the menu to download MOE analytical 
methods, upload information, read the latest 
LSB/MISA bulletins, or leave a message/comment, 

The System Operators are: 
-Gammy Lynn Raposo (Voice: (416) 235-6005) 
-David Boomer (Voice: (416) 235-5858 

Spec. BBS: (416) 235-6119) 



ONTARIO MINISTRY OF THE ENVIRONMEN 'I 

ITC SECTION 



Automation Of Both; 

Quality Control 

And 
Quality Assurance 



TOPICS & GOALS 



Topics: 

► History 

► Concepts Of QA And QC 

► Describe a tool for the 
automation of quality control, 

► Describe a tool for the 
automation of quality assurance. 

Goals: 

► Provide insight w.r.t. how we got here. 

► Highlight some important implications 
associated with how we define QA and QC. 

► Show how PC's, and PC networks can be usod 
to automate both QC and QA procedures. 



?? WHY AUTOMATE ?? 



WHY AUTOMATE ? 



QUALITY CONTROL 

► Reduce Personal Error 

► Ensure That All Procedures 
Occur As Specified 

► High Level Of Customization 
Associated With METHOD Not Technician 

► Higher Productivity 

QUALITY ASSURANCE 

► Integrate Complex Functions 

► Standardized Procedures For 
Data Review & Validation 

► Summar\y Reports From RDBMS 

->>> QUALITY IS FREE!! 



Brief History: 

- ICP X-Ray Automated Techniques; 
Generated lai ge sets of data requiring 
computation and processing. 

- 6502 Boards, CBM/PET TRS-80 etc 

- Productivity Up >300 % . . . 
-Problem/Challenge!! 

Every Time n change in DCI or QC protocol occurred we were 
faced with the 'Opirortunity To Grow' that was associated with 
reprograminiiig ALL of the different systems!! 



/ 



SOLUTION ? ! 

A NETWORK Of Computers!!! 
- A Single Program For DCI 1 ! ! 

... Also provided services like electronic mail, 
Wordprocessing, Spreadsheet, DataBase, 
Presentiition Graphics, Etc 



network:: 

QA/QC Implications 

- All Data Tlirough One Wire 

- All Data Stored In One DataBase 

- ... Obvious QA/QC Possibilities!! 



^ 



KEY CONCEPT: 

QA is Separate From QC ! ! ! 

- QC occurs in 'Real Time' and is directed at reducing 
the effect of Detemdnate Error to levels where the 
system appears to be under the influence of random 
variation; ie QC procedures should be linlted to 
known sources of determinate error. 

- QA occurs 'after the fact' and is directed at providing 
documentation tliat attests to the fact that QC 
procedures were well designed and were carried out 
in an appropriate manner. 



% 



QA is Separate From QC ! ! ! 

- Implications w^r.t. ; 

- Division Of Labour 

- Assignment Of Responsibility 

- Assignment Of Authority 

- Allocation Of Resources 

- Conceptualization Of System 



QC-EXPERT 




QA & QC AUTOMATION 



QC: { Many Custom Systems: One For Each Analytica l System T 
)► linked to sources of determinate error 
!► specific to METHO D 
^ real-time on real systems 

/ Therefore, not easily supported on LIMS 
!► "QC-EXPERT" is one example of an automated 
QC system for spectroscopic instruments. 

QA: { One Central System On PC-LIMS \ 
► links and summarizes all QC results 
!► acts after-the-fact 

>► does not haye to be customized, as with QC 
!► ILQAS is an example of an automated 
QA system 



\° 



OC-EXPERT 

» Developed in the 70's for use 

on ICP/OEB systems using PET computei^ 

> Extended during eariy 80's to ICP/I\/IS 
using PC's, dBase, and LOTUS. 

» Extended further during 80's to include 
QA database on PC network. 

• 'Collaboration' with Perkin-Elmer and 
BMB to become a commercial product. 



QC-EXPERT 

• Using a sophisticated autosampler, 
QC- Expert... 

^/ Controls the instrument & autosampler. 

V If data is out of spec, QC- Expert carries 
out predetermined procedures to restore 
quality. 

V Creates text and graphic reports 
summarizing quality control results. 

V Summary reports are sent to the LQAS 
system. 



^ 



QC-EXPERT 

Limits Can Be Associated Witii... 

Standardization Coefficients 



Coi' 



Coefficients 



QC Standards And Bianlcs 

internai Standards 

Dupiicates 

Spilces 

Diiiitions 

Sampies 







Actions May Be... 



QC-EXPERT 

Print Message And Stop 

Print l\/lessage And Continue 

Restandardize And Rerun 
Or Continue 

Restandardize And Rerun 

Wasli For Specified Time And Rerun 
Or Continue 

Rerun Current Sample/Standard 

Continue From Specified Position 

User- Defined Program 



^ 



PC BASED \AB QA SYSTEM: LQAS 



Manager 

Lotus 

dBaee 

O&A 

etc 



<C-: 



INSTRUMENTS 



L 



1} 



■=> 



Structured ASCII File 



u 



Vaxlous Parse Tools 



H 



QA SYSTEM 
And DataBase 



Expor t 





Manager 

Lotus 

dBase 

QtA 

etc 



Graphs & Repoits 
Result Approval 
Archive/Retrieval 
Methods 
Standards Dbase 



Bench Procedures 
QA Manual 
SOP'S 
Staff CV's 



Report To Client (US) 



V 




• Functions 

V Data Capture (From QC- Expert) 

V Results Approval 

V Reports And Graphs 

V View And Update; Methods, 
Bench Procedures, QA Manual, 
SOP'S, CV's, Standards Databases 
Archive And Retrieval. 

> Will run on LIMS (DOS) or 
on stand-alone computer. 



v*- 



DATA 




o 



RESULTS INSPECTION AND APPROVAL 



v'b 



RESULTS APPROVAL 



Automated QA Inspection 
And Approval... 
Report via E-Mail 

Manual QA Inspection 
And Approval 

Both Procedures Are 
Supported By The LQAS 
"Tracking System" 
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REPORTS AND GRAPH 
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HMPRECON METHOD MONTHLY QA GRAPH: Standard 
I.D.: CHK2 TEST: CDUT DATE: 91/03 
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RUN NUMBER 
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HMPRECON METHOD MONTHLY QA GRAPH: Blank 

I.D.:BLK TEST:CDUT DATE: 91/03 
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"Quality is never an accident. It is always the 
result of high intention, sincere effort, 
Intelligent direction, and skillful execution. 

It is the wise choice of many alternatives, 
requiring the combined knowledge of many 
masters of craftsmanship. 

Quality is not an attribute; it is a process, 
an Ideal that can only be realized after the 
demands of necessity have been met and aftc 
the attainment of mere usefulness" 

Willa Foster 
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SECTION THREE 



DAY 1 ~ SESSION TWO PRESENTATIONS 



CHAIRPERSONS - 



J. LEARN, DOMTAR 
C. RAPOSO, MOE 



ROUND ROBINS (MOE) - 

ROUND ROBINS (CC»>1MERCIAL LAB) 

ROUND ROBINS (WTC) - 

REGS £ JAW6 - 



S. CUSSION, MOE 

J. SZEKELY, HAMILTON REG. 
B. FOWLER, AXYS LAB 

M. FOROOTAN, WTC 
P. FOWLIE, WTC 

DR. O. HERRMANN, HYDRO 



INTERLABORATORY STUDIES 



THE MOE 
LABORATORY PERSPECTIVE 



SYLVIA CUSSION 
QUALITY MANAGEMENT OFFICE, LSB 



PURPOSE: 



Performance evaluation technique 
in support of Ministry Programs 



PROGRAMS 



MISA 



Special (N-nitrosodimethylamine) 



Air Programs 



MATRICES 



Ampouled Standards for 
Direct Instrumental Analysis 

Spiked Reagent Water 

Spiked Effluents 

PUF Cartridges 

HIVOL Filters 



PARAMETERS 

CONVENTIONALS/INORGANICS 



• Metals 

• TKN & Total Phosphorus 

• Phenolics by 4AAP 

• Oil & Grease 

• Total Cyanide 
«> Mercury 

« DOC & TOC 



PARAMETERS 



ORGANICS 



Volatiles 

Acids (Phenolics) 

Base/Neutrals 

Neutral Chlorinated 

BTX & Acrylonitrile 

N-nitrosodimethylamine (NOMA) 

Dioxin 



EVALUATION TECHNIQUES 



Statistical Summaries 

Percent Recovery of Target 

Bar Graphs - Elution Order 

Youden Plots 

Automated Graphical Procedures 
for Rapid Evaluations of 
Interlaboratory Studies 



STATISTICAL SUMMARIES 



Interlaboratory Mean 



Interlaboratory Median 



Range 



Standard Deviation 



PERCENT RECOVERY 



Use Target/Design value 

May be affected by method 
of quantitation (eg. RFA iVlethod) 
Quantitate relative to 
one parameter (eg. DHA) 



EXAMPLE OF RECOVERY 
CORRECTION - DHA 

RFA Method Quantitates all other 
parameters relative to 
Dehydroabietic Acid (DHA) 

Difference between laboratories 
attributable to difference in DHA 

Normalize parameters relative to DHA 

Re-evaluate performance of other 
parameters 
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FIGURE 11: INTERLABORATORY STUDY 88-2A 
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FIGURE 3: INTERLAB ORATORY STUDY S8-Zk 
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BAR GRAPHS - 
ELUTION ORDER 



Organic Scans 

Base order on most commonly use(i 
Gas Chromatographic Capillary 
Column 
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FIG 1: ROUND ROBIN 88-1; VOLATILES 
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SAMPLE 1 B: RE/-.GE:NT WATER LOW SPIKE 
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PARAMETER ORDER OF El.UTIOt; 
(DB-5 CAPILLARY COLl W*) 



1. 1-DICHLOROtTHENE 
METHYLENE CW.ORIDE 
t-1, 2-DICHLOROETHENf 
1. 2-DICHLOflOETHANE 
CARBON TETRACHLOniDf. 
1, 2-DICHLORQPBOPANE 

_ THICHLOROETHENE 
1 DIBHOMOCKLOnOMETHANt 
I CHLOROBENZENE 

~\ 1. 1. 2. 2-TETHACHLOROf THAh4£ 
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FIGURE 5 - INTERLABORATORY STUDY 89-1 



lij 

_l 



O 

> 

a: 

> 

o 
o 

UJ 
ft! 

H- 

Ui 

o 
a: 

UJ 

Q. 



200% -T" 
190% -I 



180% 

170% 

160% - 

150% - 

140% - 

130% - 

120% - 

110% 

100% - 

90% - 

80% - 

70% - 

60% 

50% 

40% 

30% - 

20% - 

10% 

0% 



VOIATILE ORGANICS (MISA GROUPS 16 & 17): SAMPLE VOLC 



PARAMETERS IN OROI^R OF ELUTIOK 
(BASED ON A SPB-5 CAPILURY CQUMO 

CHL0R0MEJHAJ4E BENZENE 

VINYL CHLORIDE 1,2-DICHLOROPROPANE 
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1.1-DICHLOROETHYl ENE d«-1,3-DICHL0R0PR0Fn.ENE H 
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1.1-DICHLOROETVlAHE 

CHLOROFORM 

1.2-DICHLOROETHWiE 
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DICHLOROBROMOMETHANE 

TOLUENE 

trans- 1 .3-DICHLOROPROPYLENE 

1 , 1 ,2-TRICHLOROETHANE 

ETHYLENE DIBROMIDE 

TETRACHLOROETHYLENE 

CHLOROBENZENE 

ETH^rlBE^JZENE 

m- at p-XYLENE 

BROMOFORM 

o-XYLENE 

1 .1 .2,2-TETRACHL0ROETHAr>iE 
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YOUDEN PLOTS 



Plot 1 sample on x-axis and 2nd 
sample on y-axis 

Expect results to cluster around 
target values 

Results along line passing through 
origin & target demonstrate bias 

Results in upper left & lower right 
quadrants are erratic - out of control 
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FIGURE 10 - INTERLAB ORATORY STUDY 89-1 
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SOLVENT EXTRACTABLES: YOUDEN PLOT 



IDEAL GROUPING OF DATA 
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FIG. 9: INTERLABORATORY STUDY 89-5 

BASE/tCUTRALS WITHIN LABORATORY PRECISION: L^B 2001 
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PARM4ETER RESULTS 



SAMPLE EXTB 
— TARGET PRECISION 
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AUTOMATED GRAPHICAL 
PROCEDURES 



Rapid evaluation of interlaboratory 
studies 



Poster session in TRENT //C4l/l^/?7>M 
rooms (Satiii Seiliah) 



Use to evaluate inorganics and 
organics 



FIGURE 5.6 



CHROMIUM - STUDY 89-6 



INTERLABORATORY PRECISION EVALUATION 
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INTESIABORATORY STUDY 90-1: CYANIDE 



MATRIX: REAGENT WATER 
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SAMPLE D3 VS SAMPLE D4 
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SAMPLE 04 (PPU) 
LABORATORIES 5008. 5021 it 5032 ARE OUTSIDE SCALE 
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MATRIX: LAKEVIEW STP EFFLUENT 
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SAMPLE E3co»t VS SAMPLE E4coi»r. 
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INTERLABORATORY STUDY 90-5: BTX & ACRYLONITRILE 

FIGURE 5: TOLUENE - V0C2 vs V0C4 
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ROUND ROBIN STUDIES 



JUDITH SZEKELY 

HAMILTON-WENTWORTH REGIONAL LABORATORY 



11. -l!! I«.l . <• J- III 



Fnv»rnninprit=jl labs tnda^' vary from -^ni^ll inclnstrial or novArnm'^'nt ownpfl f?;,-iiitios <o 
scpnisticateci commercial cr spscializec; gcvernmeniai ssi ups. fr^ey a;! prccucs 
numbers which have an impact on industries, local economy and the community. 



* Lack of a regulatory body who enforces: 

GLP. 

Universal methods. 

Establish acceptable standards of instrumentation. 

* Small facilities with not always qualified personnel and very limited resources. 

Lack the possibility of extensive QA/QC. 
Method development. 

All these facilities with a large range of possibilities produce data. 

For this data to have any meaning, in other words, to lend them consistency and a level 
of confidence need arises for a regulatory body. All ihese labs have to be brought to a 
common denominator. 

We do not have that regulatory body at this time, and it seems to me that nobody really 
wants to assume this responsibility (not even, and especially not our gracious hosts, right 
here, the MOE), Different agencies cover part of the activities, such as CAEAL for 
certification. ACPO, licencing of the chemical professionals, etc. 

We find under the present situation a solution to these problems in running Round Robin 
studies. 

I guess we were not the only ones to discover that, so what happened - just like with any 
other commodity, round rotiiins were set up by chemicals suppliers. CAEAL and others. 

'.Ve a.'c ai tae poin; now ir.ai it l5 cifficul'. to choose ihe right siuay lo p£r:al\e 's,. 

For a round robin studv to be efficient it should satisfv the followino reouirementS' 

* Samples have to be certified, with all the QA/QC parameters defineo m 
advance (detection limit, range, assay value, and method used to determine 
it). 

* A set of samples should include: 

Standards. 
Test samples. 
Spikes. 



?;vRilobl%- otherwise results are misleadma. 



Communication nemork - to coorainaie ana give aavance nonce of sample 
arrival to ensure laboratories are able to handle it. 

* The "body" coordinating the study should have proven credibility. 

Taking part in a round robin, might become fairly expensive and laborious, and quite a 
nuisance; since we are not quite regulated in this field why bother would many 
managers/owners ask. 

Being part of such a study has great advantages- 

* Raises confidence in the quality of the laboratories results, lends credibility 
to the facility. 

* No matter how good the laboratories own QA/OC program can be. there 
might be a systematic error producing consistently erroneous results for a 
given parameter. 

* Will provide valuable support data in court litigation. 

At this point we arrived to MISA. Up to here, we outlined the QA/QC aspect of round 
robins m ensuring that quality results are supplied to the enforcement personnel 

There may be another aspect to it, though. MISA provides for the municipalities not only 
to monitor industries through their own tests, but also by accepting documentation 
provided by industries, as to their 'good" performance. 

The industries, on the other hand, can produce this documentation by using certified 
commercial labs or their own facilities. In this second case, these facilities should be 
asked to demonstrate the validity of their documentation: 

* l-'dustrial lab should be inspected for GLP. documents. OA/QC. 

* Participation in round robin studies, and proof of quality of results should 
also enable industries to suoport their results. 



ROUND ROBIN STUDIES 



BRIAN FOWLER 

AXYS ANALYTICAL LABORATORY 



ROUND ROBIN 



DEFINITION: A STUDY IN WHICH ONE OR 

MORE SAMPLES ARE ANALYSED 
INDEPENDENTLY BY SEVERAL 
ANALYSTS FOR THE SAME 
COMPOUNDS. 
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ROLE OF ROUND ROBINS 
IN THE EVOLUTION OF A METHOD 



METHOD DEVELOPMENT 
(Standard recovery, matrix spikes 
linearities, environmental samples) 




^ 



jj^ 



Preliminary Method 



::Ll. 



Method Intercalibration 
Round Robin (internal) 



Routine Analysis 
^ 



Other Labs 
(1-3) 



\L 



Reference Samples 



^ 



Lab Evaluation 
Round Robin (External) 



Feedback 



Other Labs 
Many 



Lab Performance Data 




j£ 



Consensus Reference 
Materials 



THE PROS AND CONS 

OF ROUND ROBIN EXERCISES 

FOR PRIVATE LABS 



PROS — THE BENEFITS 



1. Detect systematic method problems. 

2, Detect errors in analytical standards. 

3, Independent analysis can provide greater confidence in 
data. 

4, A high ranking may benefit a lab and may serve as a 
promotion for the analyses involved. 

5. Potentially could generate more Reference Materials. 

6. The exercise can validate a developed method. 



CONS — THE DISADVANTAGES 

1. Round robin work may conflict with work load. 

2. Added costs and commitment of staff and resources. 

3. There is the possibility that a low ranking may work 
against a lab when not at the top. 

4. Round Robins emphasise "consensus values" which may 
not be the 'design' or correct values due to common 
interferences or method differences. 



PROBLEMS WITH ROUND ROBINS 



1. No feedback report from submitted data, 

2. No detectable analytes in the round robin 
samples. 

3. Client-supplied surrogates contaminated with 
a target analyte and allowed no mechanism 
for blank correction. 

4. Confidentiality not assured. 

5. Data may not be evaluated in as an 
anonymous set. 

6. Calculation methods may bias results 
especially for detection limits. 

7. *Consensus value may not be 'correct'. 



SUGGESTIONS FOR 
MORE USEFUL ROUND ROBINS 



1 . Round Robins are most useful to participating labs when there is 
timely feedback. A detailed discussion and comparison can 
come later. 

2. The reporting requirements and evaluation criteria should be 
clearly laid out in documentation provided to participants. 

3. Analyte levels in Round Robin samples should preferably not be 
at detection limits but at least ten times typical detection limits. 
Required detection limits should be stated. 

4. Round Robins work well for single analytes but are less reliable 
for group totals. The problem of group totals may need to be 
evaluated using different criteria than for single component 
analytes. 

5. When no SRM/CRM is available comparable with the round robin 
sample, sufficient sample should be prepared to allow it to be 
used as a "consensus" reference material. 

6. Round robins should be scheduled for less busy periods. 
Perhaps a questionnaire could identify a preferred period. 

7. Ranking participants is contentious; there will always be winners 
and losers when all may be qualified, (or conceivably none are 
qualified). Perhaps a better way may be to classify labs as 
"currently qualified" for a specific analysis. 
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ROUND ROBIN STUDIES 



- DESIGN, IMPLEMENTATION AND VALUE 
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QUOTE FROM A FAMOUS CHEMIST: 

All you have to do to get the right answer is to send the 
sample to dozens of labs around the world, throw out 
the outliers, and take the median. That will be close to 
the truth. 

Claude W. Sill 
US Dept of Energy 
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WHAT IS A ROUND ROBIN 



* Several identical sets of Perform Evaluation samples sent to 
a group of labs for analysis 



Results are analyzed, and a report prepared 



An experiment designed to achieve an objective 




WHAT CAN A ROUND ROBIN DO 



Provides useful information on variability in measurements 
within labs 



* Identifies the sources of variability in datasets 



* Provide feedback on performance to iabs and to others 



* Identifies weak laboratories and can provide clues to the 
source of the problem 




I 
I 



WHAT INFORMATION CAN BE PROVIDED 

* Performance of different methods 

- specify methods exactly 

- bias, precision, ruggedness 

- methods failure hypotheses 

- interference assessment 

- assessment of useful range and scope 

* Snapshot of Data Quality as it exists with existing methods 

* Comparison of the instrument 




WHAT INFORMATION CAN BE PROVIDED 

* Identify specific problem areas 

- calibration curves, series of standards 

- instrument (injection ready) 

- method 

- blanks, extrapolate to zero 

- matrix effects, series of sample spikes 

* Comparison of groups of labs 

- Commercial vs Non - Commercial 

- Canada vs others 

- individual lab compared to the group 

- practical multi lab MDL (POL) assessment 




PQL 

practical quantitation iimit 

EPA approach 

Concentration above which 80% of labs 
can get within 40% of true value 

Useful for multi-lab datasets 

Reflects differences(biases) due to 

instrument, stds, analysts, methods 




1 

1 






Figure 13 - Percentage of Laboratories 
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DESIGN i 

* range 

* high and low duplicate 

* matrix (natural or synthetic) 

* blank where applicable 

* stability 

* homogeneity 

* interference 

* contamination 

* loss 
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HOW rO 00 A ROUMD ROBiN 

* Identify labs that are willing to participate 

* Prepare samples according to design 

- Usually of known composition 

- Standards, blank, real samples 

- Reference materials characterized by a number of labs 

- bottled in large runs 

- Homogeneity of ail bottles 

- Stability by preservation & controlled temperature 

- Make sure there is no contamination or loss 

* Ship the samples as soon as possibie 
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WASTEWATER TECHNOLOGY CENTRE 



is both a participant in, and a provider of round robin 
studies 

uses round robin data as part of internal QC activities 

uses round robin data to validate the samples used 

provides quick feedback to participants in WTC 
operated round robins 

has done AOX, SS, BOD, pH, P, CN 




TABLE 2 
ANALYTICAL RESULTS 



SAMPLE 1 


2 


3 


4 


5 


6 


LAB 












1 


259 . 


7.3 


3.9 


40.0 


8.1 


4.0 


2 


213.0 


6.1 


3.0 


33.5 


7.7 


5.0 


* 3 


679.0 


620.0 


2.1 


236 . 


9.6 


3.7 


4- 


257.0 


7.3 


1.8 


40.0 


8.0 


4.2 


* 5 
6 

7 


209.0 


13.8 


11.3 


42.5 


31.3 


16.3 


202.0 


5.7 


2.6 


29.4 


7.1 


3.7 


8 


194.0 


8.8 


3.6 


35.6 


6.8 


5.3 


9 


229.0 


5.9 


Z.3 


33.1 


6.5 


2.9 


10 


247.0 


6.2 


3.5 


37.0 


8.5 


3.7 


11 


251.0 


7.2 


3.8 


38.4 


7.8 


3.8 


12 


208.0 


6.1 


3.1 


34.4 


7.1 


3.6 


13 


203.0 


4.6 


2.4 


30.5 


7.4 


4.1 


14 


186.0 


6.7 


3.1 


30.0 


7.1 


3.4 


15 


214.0 


5.9 


2.9 


37.3 


7.3 


3.7 


16 


220.0 


5.2 


2.4 


30.0 


6.9 


3.3 


17 


209.0 


4.5 


2.7 


30.1 


6.3 


2.7 


18 


209.0 


5.1 


2.3 


29.8 


7.5 


3.5 


19 


198.0 


3.7 


1.8 


28.1 


6.9 


3.6 


20 


204.0 


5.3 


2.8 


29.0 


7.6 


4.0 


21 


207.0 


5.1 


2.7 


31.1 


7.7 


4.0 


22 


212.0 


5.3 


2.7 


28.2 


7.8 


4.0 


23 


245.0 


6.1 


3.2 


38.0 


8.8 


3.8 


24 


170.0 


5.2 


2.8 


25.0 


7.0 


2.3 


25 


250.0 


4.8 


1.9 


34.0 


8.0 


3.4 


26 


286 . 


5.0 


2.7 


35.0 


7.4 


3.9 


MEAN 


221.0 


5.8 


2.8 


32.9 


7.4 


3.8 


MEDIAN 


212.0 


5.7 


2.8 


33.1 


7.4 


3.7 


STDEV 


27.7 


1.1 


0.6 


4.2 


0.6 


0.6 


c.o.v 


12.6 


19.5 


20.6 


12.7 


8.2 


15.1 


MIN 


170.0 


3.7 


1.8 


25.0 


6.3 


2.7 


MAX 


286.0 


8.8 


3.9 


40.0 


8.8 


5.3 


♦ DATA NOT USED IN STATISTICAL CALCUUTIONS 
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Fig. 12 An Exampit of a Statistical Outlier (Grubbs) 
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Fig. 11 Evaluotlon of Dota bated on 10X 4 1 Std. D«v. Rule 
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Fig. 2a Some t/pTcal types of Laboratory Performance 
revealed by External QA Studies 
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Fig. 2b Soma typical typ«t of Loborotory Parformonce 
rtvecled by ExttrDol QA Studies 
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Laboratory: W0072 

Comoarison of Results RsDorted versus 
the Interlaboratory Median values 
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Laboratory: W0506 

Comparison of Resuiis Reported versus 

ths Inisrlcboratory Ivledlan vGlues 
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Lab Code: W0506 




Laboratory: W0489 

Comparison of Results Reported vers'j: 
the Inferlaboraforv Median values 
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Laboratory: W040^ 

Comparison of Results Reporied versus 

the Inieriaboraiory r/ledlar. vg!u = 3 
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Laboratory: W0198 

Comparison of Results Reported versus 

the Inferlcboratory Wedlcn vciuss 
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WHAT USE IS A ROUND ROBIN 

USES FOR PARTICIPANTS: 

- External QC, compare to peers ^ 

- feedback on data quality produced 

• can identify and diagnose problems before your clients 
Identify them for you 

- due diligence In your QC efforts 

- evaluate your lab mettiods 

- useful info to educate clients 
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USES FOR PROVIDERS OF ROUND ROBINS: 

- Info on performance Of the lab community 

- Info on specific labs 

- usually confidential 

- useful for contracting analysis 

- Info on expected quality of a multilab database 

- Info on method performance 

- Data on 'True Value" of a reference material by a selection 
of methods 

- Get really good methods specifications on your own lab 




USES FOR CONSUMERS OF DATA: 

- individual lab performance 

- need to know basis 

- expected database quality 

- Can realistically identify DQO*s and project plans 

- Can identify trends in data quality 

• Can force Improvements in data quality 




USES FOR REGULATORS AND DECISION MAKERS: 

- To ensure that decisions are made on data of known 
quality and that the limitations and variability in the data 
are taken into account 
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ANALYTICAL CHEMISTRY UNDER REGULATIONS 



OTTO HERRMANN 
J.A.W.G. 
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INDUSTRY MOE 



"IDEAL" METHOD ' LIMITED TIME & RESOURCES 

+ i" 

EXTERNAL LIMITATION POLITICAL INTERFERENCE VS 

(SPECIFIC METHOD & VARIATION) TECHNICAL JUDGEMENT 

+ + 

LEGALLY RESPONSIBLE MULTIPLE INTERPRETATIONS OF 

+ "UNAMBIG^JOUS" DESCRIPTION 

INTERPRETATION 
(WRITTEN BY LAWYERS) 

+ 
TIME LIMIT 



COMMON GOALS 

BUT 

LOCAL STRESSES ====> 

POLARIZED VIEWS 



I 
I 
I 



I 




Analytical 
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• INDUSTRIAL SECTORS 
Inorganic Chemical 
Iron & Steel 

Metal Casting 

Mining & Industrial Minerals 
Organic Chemical 
Petroleum Refining 
Power Generation 
Pulp & Paper 

• Environment Canada (Wastewater Technology Centre) 

• Ministry of the Environment 

• "VISITORS" 
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PURPOSE: DEAL WITH ANALYTICAL ISSUES 



GENERALLY TECHNICAL REPS VS ADMINISTRATORS 
WIDE REPRESENTATION BECAUSE ISSUES CUT ACROSS SECTORS 
PROVIDES "ONE-STOP SHOPPING" ON ANALYTICAL ISSUES 
"VISITORS" TO DISCUSS SPECIAL CONCERNS OR TO PROVIDE 
EXPERTISE FROM BOTH MINISTRY AND INDUSTRY 
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so WHAT'S HAPPENED? 



• FORMATION PROPOSED BY DON KERR (CONSULTANT) AUGUST 1990 

m 

• 1ST MEETING HELD OCTOBER 1990 



• 12TH MEETING WILL BE IN MAY 1992. 
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A MEETING IS NOT AN "EVENT" IN ITSELF 

WHAT'S BEEN DISCUSSED? 



1. REHASH THE ANALYTICAL AND SAMPLING PROTOCOL DOCUMENT 

2. QA/QC - REQUIREMENTS. CHARTING. OUT OF CONTROL 

3. LMDL - HOW AND WHEN 

4. NIMMP (NEW INSTRUMENTAL MEASUREMENT METHOD PRINCIPLES) 

5. RECORD KEEPING AND DATA RETENTION 

6. MIDES AND DATA ENTRY - PROBLEMS AND REVISIONS 

7. LABORATORY WORKSHOPS - THIS IS IT! 

8. HEXACHLOROCYCLOPENTADIENE (IN ATG23) - FORGET IT 

9. PHENOLIC CONTAMINATION IN AUTOSAMPLERS - WATCH IT 

10. PHENOLIC PRESERVATION - ATG14 (4AAP PHENOLICS) & ATG20 

11. DIOXINS 

12. ION CHROMATOGRAPHY FOR FLUORIDE AND SULFATE - SURE, WHY NOT? 

13. SOLVENT EXTRACTABLES (ATG25; OIL & GREASE)) 

14. VCM/BTX (VINYL CHLORIDE MONOMER. BENZENE. TOLUENE. XYLENE) 

15. TOTAL PHOSPHORUS AND STANNOUS CHLORIDE - DONT DO IT! 

16. ANTIMONY AND FALSE POSITIVES - CANT PRECHARGE PET BOTTLES 

17. ALKYL LEADS (ATG13)- GRAB SAMPLING ONLY 

18. STYRENE AND 0-XYLENE - CHROMATOGRAPHIC RESOLUTION RESOLVES 
THE PROBLEM 
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PRESERVATION OF PHENOUCS 



• HOW IT WAS (IS): 

SULFURICACIDTOpH 1.5-2: 10 DAYS 

PHOSPHORIC ACID / COPPER SULPHATE PRECHARGED; 4 DAYS 

• MULTI-LAB STUDY (MINISTRY PLUS INDUSTRY) 

1. STANDARDS IN DEIONIZED WATER + PRESERVATIVES => OK 30+ DAYS 

2. REAL SAMPLES "AS IS", SPIKED SAMPLES + PRESERVATIVES ■ 
STUDY JUST FINISHING 

LOOKING HOPEFUL 

• BUT 

WHAT ABOUT ATG20? NO PRESERVATIVES FOR 30 DAYS. 
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SOLVENT EXTRACTABLES (ATG25) 



• GRAB VS AUTOSAMPLER - OBVIOUS PHASE SEPARATION? 

• COMMON METHODS USE: 

CHLOROFLUOROGARBON (FREON 113) 
DIGHLOROMETHANE (DCM) 

• BUT: 

EXTRACTION EFFICIENCIES NOT SAME - WHAT WILL BE REGULATED? 

CFC USE BANNED IN 1994 

MUCH HISTORICAL DATA BASED ON FREON 

MUNICIPALITIES SPECIFY FREON 

EPA CONSIDERED HEXANE / METHYL TRIBUTYL ETHER (20/80) 

DORMANT 
EUROPE JUST ABANDONED CCL4 FOR FREON 

• INTERIM: 

TENDING TOWARD DIGHLOROMETHANE 
RECOMMEND NOT REGULATE BASED ON FREON DATA 
POSSIBLE MONITORING PHASE BEFORE SET REGULATORY LIMIT 

• INVESTIGATION OF SOLID PHASE EXTRACTION ALTERNATIVE 

• UNRESOLVED: WHY? WHAT IS THE PURPOSE? 
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DATA ENTRY & MIDES 



» PROBLEMS:DIFFICULT ENTRY 

ORIENTATION TO SMALL MANUAL OPERATION 
REVISIONS DESTROY BUG-FIXES 
DIFFICULT COMMUNICATION 

• WHY JAWG AS THE INTERFACE? 

TECHNICAL PROBLEM 

COMMON TO ALL 

SINGLE INTERFACE BETWEEN MINISTRY AND INDUSTRY 

• WHERE ARE WE? 1 

I 
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Y & THE FUTURE 



• THE PeOCESS WORKS 



• IMCORPORATE MUNICIPAL SECTOR • IN PROCESS 

• COMMERCIAL LABS ■ BEGIN PROCESS OF REPRESENTATION 



• NEEDED MINISTRY IMPROVEMENTS 

PRESENT UNIFORM, EASY-TO-ACCESS FRONT 

BETTER PROCESS FOR ACTION ONCE CONSENSUS REACHED 

SORT OUT THE INTERNAL BATTLES FOR POWER 

• QUOTABLE QUOTES 

'IFTHEREGUWTiONSDONTFORBIOPOIT' 

"IT'S EASIER TO GET FORGIVENESS THAN PERMISSION' 

IF THE EDICTS COME OUT WITH ONE WBEL, SO SHOULD THE SOLUTIONS' 
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SECTION FOUR 

DAY 2 - SESSION THREE PRESENTATIONS 

CHAIRPERSONS - 6. STEINKE, MOE 

R. CLEMENT, MOE 

PROTOCOL DOCUMENT - P. BAULU, MOE 

NON-COMPLZANCE/ElirORCEMBMT - C. DOBHLER, MOE 

DETSCTION/LOH LEVEL DATA - D. KING, MOB 

CAEAL - B. TRAVSRSY, CABAL 

6. CRAWFORD, MOB 



PROTOCOL FOR THE SAMPLING AND 



ANALYSIS OF INDUSTRIAL/MUNICIPAL 



WASTEWATER 



SAMPLING/ANALYTICAL PROTOCOL 

Introduction 

• PROTOCOL WILL BE REFERENCED IN LIMITS 
REGULATIONS 



• WILL BE CONSIDERED AS A LEGAL DOCUMENT 

• REQUIREMENTS FOR SAMPLING AND ANALYSIS 
OF EFFLUENTS UNDER LIMITS REGULATIONS 

• EASE OF REVISION/AMENDMENT 

• MAY BE REFERENCED IN OTHER PROGRAMMES 
(SUCH AS CERTIFICATES OF APPROVAL) 



SAMPLING/ANALYTICAL PROTOCOL 

Introduction 

• REGULATION REQUIREMENTS EXPRESSED IN 
TECHNICAL/SCIENTIFIC LANGUAGE 



• RECOMMENDATIONS & SUGGESTIONS INCLUDED 

• ADDITIONAL TEXT FOR CLARIFICATION OF MOE 
INTENTIONS & EXPECTATIONS 

• FOR EASY REFERENCE, ATG GUIDE LISTS ALL 
REQUIREMENTS FOR THAT GROUP OF PARAMETERS 
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SAMPLING/ANALYTICAL PROTOCOL 

Introduction 



OUTLINES PRINCIPLES TO BE FOLLOWED FOR: 

• SAMPLING 

• PRESERVATION 

• STORAGE 

• ANALYSIS 

• QUALITY CONTROL 

SEPARATE DOCUMENT OUTLINES DATA REPORTING 
PROTOCOL 



SAMPLING/ANALYTICAL PROTOCOL 

Data reporting 

• DATA PROTOCOL LISTS MINIMUM REQUIREMENTS 
FOR COMPLETE DATA BASE 

• REPORTING REQUIREMENTS ARE EXPECTED TO 
BE STATED IN THE REGULATIONS 

• REPORTING REQUIREMENTS MAY BE MINIMAL WIT 
BULK OF DATA RETAINED BY DISCHARGER FOR 
MOE REVIEW UPON REQUEST OR 

SUBMITTED TO MOE AT SPECIFIED FREQUENCY 
QUARTERLY ?) 



SAMPLING/ANALYTICAL PROTOCOL 

ATG guide 



• SECTION 5, ATG GUIDE, SET-UP FOR QUICK 
& EASY REFERENCE 

• TABLE FOR EACH ANALYTICAL TEST GROUP (ATG) 



• EACH TABLE CONTAINS ALL INFORMATION FOR 
SAMPLING, ANALYSIS AND QC FOR THAT TEST 



SAMPLING/ANALYTICAL PROTOCOL 

Sampling 

REQUIREMENTS LISTED FOR: 
SAMPLING TECHNIQUES 
SAMPLE CONTAINERS 
CONTAINER PRE-TREATMENT 
PRESERVATION 
STORAGE 

RECOMMENDED SAMPLE VOLUME 
PRECAUTIONS & REMARKS 
USER'S CHECKLIST IN SCHEDULE 2 



S AMPLING/ANALYTICAL PROTOCOL 

Analysis 



OUTLINES ANALYTICAL PRINCIPLES FOR: 



SAMPLE PREPARATION & PRE-TREATMENT 

ANALYTICAL TECHNIQUE 

INSTRUMENTAL MEASUREMENT METHODS 

REPORTING UNITS 

PERFORMANCE CRITERIA/RMDL 

ALTERNATE & NOT RECOMMENDED METHODS 
ALSO LISTED WHERE APPLICABLE 



S AMPLING/ANALYTICAL PROTOCOL 

Quality control 



REQUIREMENTS FOR TYPE & FREQUENCY OF LAB 
& FIELD QC SAMPLES: 

LAB QC SAMPLES INCLUDE : METHOD BLANKS, 
SPIKED BLANKS, SPIKED SAMPLES & REPLICATES 

RECOMMENDED FIELD QC SAMPLES : TRAVELLING 
BLANKS, TRAVELLING SPIKES & DUPLICATES 
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Presenting 



Potential Non-Compliances/ 

Enforcement 



Cathy Doehler 

Laboratory Services Branch 

Ontario Ministry of Environment 



Non-Compliances/Enforcement 



inspection types/personnel 

lab compliance items 

report preparation 

follow-up action/personnel 

MOE compliance policy 

LSB services/lab responsibilities 



MOE Inspection Personnel 



Regional staff 

(MISA and/or Abatement) 

Water Resources Branch 

(MISA Industrial, toxicity, flow monitoring) 

Laboratory Services Branch 

Investigations and Enforcement Branch 



Inspection Types /Personnel 



spot 
(regional staff) 

routine 
(regional staff) 

detailed 

(region, LSB, WRB) 



Inspection Areas 



site 

flow 

sampling 

analysis 

toxicity testing 

data reporting 



Laboratory Inspection Areas 



sampling/preservation 

sample storage/transport 

sample analysis 

QA and control limits 

data reporting and interpretation 

staffing and safety 



LSB INSPECTION PROGRAM 



• 76 industrial labs 
26 commercial labs 



inspections prioritized 
-number of parameters analyzed 
-method complexity 
-location 



Laboratory Compliance Items 



use of SRMs 

bench QC 

demonstration of control status 

mdl calculations 

calibration curves 

temperature maintenance 



FUTURE LSB INSPECTIONS 



emphasize compliance items 

data consistency/improvement focus 

final details in development 



GLP Suggestions 



sampling/ 

preservation recommendations 

balance weight checks 

record keeping improvements 

control charting suggestions 

housekeeping improvements 



COMPLIANCE 



"a state achieved by adhering to 

the legislative and regulatory 

requirements of the Ontario 

Ministry of Environment" 



MOE compliance policy 



Inspection Results 



• prepare reports 
-commercial lab/LSB 
-regional Inspection 

• follow up action 



LSB will continue to: 



• perform lab inspections 

• prepare inspection summary reports 

• consult/advise regions/industry /labs/WRB 
on MISA-related laboratory concerns 

• provide LSB MISA methods (BBS available) 

• organize and manage interlab studies 



LSB will not. 



• approve/certify laboratories 

• recprnmend commercial labs 

• approve use of non-regulation methods 

• provide standards or check samples 

• lay or proceed with charges 



MISA 



"virtual elimination 

of toxic and persistent 

contaminants" 



LOW-LEVEL DATA ISSUES 
MISA PROGRAM 



I 

MISA WORKSHOP 1 992 



Don King 
Environment Ontario 
Laboratory Services Branch 



MISA WORKSHOP 1992 

LOW-LEVEL DATA ISSUES 

E>on King 

DISCUSSION 

This presentation shows diagrammatically the relationships among: performance criteria, perfonnance 
indicators, low-level measiirement practices, and data reporting practices. It also reviews the process for 
setting compliance limits based on MISA effluent data including th& Long-Term Average (LTA) and the 
Variability Factor (Vf). It discusses the effect of low-level data censoring on the Limit Setting process. 
It also discusses the interpretation of data that falls above the compliance limit 

Fig.l geft side) 

For the MISA Monitoring Regulation laboratories were required to demonstrate their ability to measure 
at low-levels. For this purpose the regulation specified an upper limit for the laboratory's method 
detection limit (MDL) and required all laboratories to estimate their MDL using a specified protocol 
since revised to address multi-analyte scan situations. For the purposes of this discussion the criteria value 
is RMDL, and the individual laboratory estimate is LMDL. In order to estimate LMDL the lab must 
calculate the within-batch standard deviation (Sw) for eight replicate analyses of a low-level spiked blank 
and multiply this value by 2.998. 

Fig.l (middle) 

To ensure a reliable estimate of Sw, the lab is essentially required to be able to read results to the nearest 
RMDL/10. For MISA this value is labelled as W. It represents both a reporting increment (e.g when 
RMDL = 20 then results should be reported in steps not larger than 2, thus 18, 20, 22. 24 ...), and a lowest 
reportable value. When a measured result is below W, then it would be reported as 2 <W. The remark 
code <W signifies essentially a null response. For many labs the MISA W value falls between Sw/3 and 
Sw. Two other decision points were recognized. The code DL is set at the LMDL value and results below 
DL were coded <DL. This remark signifies that the presence of analyte in the sample is not entirely 
demonstrated. T is the code for the RMDL value, it represents 10 times W. Results below RMDL but 
at or above DL would be coded <T. This signifies a 'tentative' resulL 

Note: While a result coded <DL can be obtained even if the analyte is not present in the sample, the 
sample may stiU contain analyte at levels up to the reported value plus LMDL . 

Note: Inability to measure a specific sample because of matrix effects required the lab to indicate an 
upper limit for the possible analyte level. Such data would be coded <. 

Fig.l fright) 

For all sectors except Petroleum, laboratories were required to report results down to their LMDL value. 
They were permitted to report lower-level data. During the effluent data review, all results below 
RMDL/10 were equated to RMDL/10. The availability of this low-level data assisted greatly in the QA 
review of the effluent data. It provided better comparison betweai low-level effluent values and the 
corresponding travelling blank data, with the result that some parameters were 'deselected' as candidates 
for Umit-setong. (Real data is easier to interpret than 'less than' data.) 



Fig.2 (top) 

Tentative effluent limits arc estimated based on the long-tcrai average (LTA) and variability (V{) for the 
effluent data. The concentration limit is defined as LTA times Vf where Vf is the 99% confidence factor 
used to identify the point beyond which a result is taken as strong evidence mat the acmai conccuLrauun 
today exceeds the LTA value. For MISA, limits will be specified for loadings (concentration times flow). 
If a <T, <DL, or <W result is used to estimate loading, then the loading estimate should include a similar 
remark. For those parameters subjea to effluent loading limits such data might not be irrelevant, smce 
very high flows may combine with very low concentration estimates to cause an apparent exceedence. 

Even though the Ministry may not that the company has failed to maintain an acceptable effluent quality 
until the 99% confidence value has been exceeded, the company should recognize that effluent quality 
might be a concern whenever the LTA value is exceeded. Thus if the value exceeds LTA + (Vf-l)/3 the 
confidence level is already approaching 80% that effluent quality is degraded. 

Note that the Vf value will generally be much larger than the corresponding LMDL. The Umiis arc based 
on a Vf derived from single estimates. It includes all sources of laboratory, sampling, and industry 
process variability. It may be best to perform only one analysis, reserving the option to repeat the 
measurement for only those cases where the measurement process is itrelf suspect. This includes evidence 
of poor recovery, poor blank estimates, etc., without reference to the impact of the result on the client. 
If laboratories perform replicate analyses whenever there is a risk that their client's effluent might be 
deemed out-of-compliance, care may be required in reporting the data. Thus should the lab report the 
highest, the lowest, the middle, or all values? A single estimate may be the safest approach? 



Initially there was concern that inclusion of low-level data would lead to low estimates of LTA which 
would tighten the effluent quality criteria. It is now recognized that even though the LTA may be lower, 
the Vf value is generally much larger, so that the 'limit' is higher than if data is censored. In addition, 
the inclusion of low data often brings the LTA below the RMDL value. As a general principle such data 
is not considered sufficiently reliable to permit calculation of defensible limits. As noted eariier, such data 
was often siraUar to the travelling blank estimates and therefore led to deselection of the parameter for 
limit setting purposes. 

Summary 

The designation of an RMDL value ensured that all labs were on a level playing ground in teims of 
measurement capability. Actual sample matrix effects may forestall the analysis of the sample. And often 
matrix effects may cloud the interpretation of measurement data. But the availability of low level data 
has had a distinct positive effect on the evaluation of effluent data quality, and on the use of this data for 
possible limit setting purposes. 



1) Estimation of Analytical Method Detection limits, ISBN 0-7729-88 17-x, (rev. June 1991) 

MISA document available fiom Communications Branch, MISA Office, or Lab Services Branch 
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LOW-LEVEL DATA ISSUES: MISA 



1. Adequate Sensitivity Regulation MDL 

2. Demonstrated Ability Laboratory MDL 

3. Consistent Reporting Increment W 

4. Specific Decision Points DL T 

5. Low-level Data Qualifiers <T, <DL, <W 

Inability to measure <, ( <WE ) 

6. Censored and Non-Censored Data 
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Umit 



Vf.LTA 



LTA 




If Result above Umit 



Sw J 
'ZERO*- 



then we conclude 
that effluent quality 
exceeds the Limit LTA 



<^ 



Reg MDL 

lab A 
Lab MDL '^''^B 




how much confidence 
is required ? 

99% confidence 
95% ? 



80% ? 



NOTE: 

Vf includes all sources 
of variance 

Vf.LTA >> RIWDL ?? 
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EFFECT OF CENSORING ON LIMITS 
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L OW-LEVEL DATA ISSUES: MISA 

USE OF DATA 

1. < Results = Unable to Measure 

Amount present unknown 

2. <T Results = reported measurement below RMDL 

3. <DL Flesults = reported measurement below Lab MDL 

4. <W Results = measurement was below RMDL/IO 

5. Compliance decided on loadings > limit 

6. Virtual Elimination versus Zero Discharge 
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L OW-LEVEL DATA ISSUES: MISA 

SUMMARY 

1. Loading Estimates should carry over Lab Remark Codes 

2. For Effluent Compliance purposes Data < RMDL is not used 

3. For Effluent Assessment purposes Data < RMDL is used 

Loadings based on <W Data interpreted as 'Zero' 
Loadings based on <DL, <T Data used as Calculated 

Loadings should not be calculated from < Data 

4. Final decision not yet made on reporting <DL data 

5. Replicate Measurements if Near Effluent Limit ? 
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1. Why formed? 

2. By Whom? ' 

3. What are its objectives, mission? 

What is its structure and what are its 
programs to meet the objectives? 

Where are we now? 



THE CANADIAN ASSOCIATION FOR 
ENVIRONMENTAL ANALYTICAL LABORATORIES 

CAEAL" 




ACLAE 



f Why was CAEAL formed? 



n concerns had been raised about data being 
produced by analytical Laboratories in the 
environmental field 



^ this problem had been compounded by 
government policies 
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11 data quality problems were noted by various 
agencies during contractincj out initiatives 

■ increase in the number of private labs eacli 
with its own QA/QC procedures or none at 

! 

all 
m fly-by-night labs/low bids 
m IJC wrote: "Millions of dollars are being 

* * * 

wasted" 
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W'HY SUCH CONCERN ABOUT POOR DATA? 




The data produced by laboratories is the 

starting point for the development of iegisiation, 
government policies and regulatory measures 
designed to protect the environment Poor data 
can result in costly mistakes and affect our 
decisions on protecting the environment. 
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BY WHOM? 



In 1988, a group with representation from 
industry, government and universities was 
requested to consider the value of forming an 
Association through v*/hich a national QA/QC 
program could be developed for the self- 
regulation of laboratory services. 
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For industry, the lack of a national QA/QC 
program was seen as a strong negative 
influence on competitiveness in the 
industry sector (national and international) 

Governments were all for it, since they 
would be the main benefactors of such a 
program 
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THE CANADIAN ASSOCIATION FOR 
ENVIRONMENTAL ANALYTICAL LABORATORIES 

CAEAL 




n Governments would not need to "police" 
the industry as some had suggested 
should be done 
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MISSION 




CAEAL was incorporated in June 1989 with 
tlie following mission: 

to improve the quality of laboratory 

data, in order to develop effective 
policies and regulations for 
legislators and decision makers to 
protect Canada's environment. 
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To Improve laboratory analytical quality, 
through the provision of a national 
laboratory certification/accreditation 
program 

To provide a national forum for 
communication and dialogue among 

laboratories 
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THE CANADIAN ASSOCIATION FOR 
ENVIRONMENTAL ANALYTICAL LABORATORIES 

CAEAL" 




To help the industry upgrade Its product 
and competitiveness. 
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1 CAEAL OPERATING STRUCTURE 

a) STRUCTURE 

The management team consists of a Board of 
Directors responsible to tlie membership. The 
board is composed of eight voting members and 
meets about three times yearly. 

A secretariat provides support to the Board and 
each of the program committees. 
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An Advisory Panel recommends to the Board 
re: certification or suspension of certification. 

Eacli program area is under tlie direction of a 
program committee appointed by the Board. 

The day to day operation is handled by a 
Program Manager and an Office Assistant. 
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b) FINANCIAL CONSIDERATIONS 

CAEAL is funded by membership fees and 
fees charged to laboratories seeking 
certification. CAEAL has received financial 
support and continues to receive in kind 
support from governments and private 
industry. CAEAL operates on a cost- 
recovery basis. 
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c) CANADIAN COUNCIL OF MINISTERS OF 
THE ENVIRONMENT (CCME) 

The CCME has given strong support to 
CAEAL. It has instructed their members 
right across Canada to use CAEAL 
Accredited laboratories. The CCME 
encourages laboratories in each province 
and territory to become members of 
CAEAL 
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d) PROGRAMS "^ QA/QC 

CAEAL has, as its principal objective, the 
promotion and maintenance of a strong 
environmental analytical service within Canada. 
A vehicle for achieving the objective is provided 
by the Laboratory Certification/Accreditation 
program offered by CAEAL. 
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I ASE I : CEmWICA TION 



Certification is tlie formal recognition by the 
Association of tlie proficiency of an environmentai 
analytical laboratory to carry out specific tests. 
Under this program, participating laboratories are 
tjent test samples at six month intervals for 
analyses. 
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II 10 certification granted relates solely lo the specific 
lefts for which a laboratory seeks certification and for 
which certification Is available. 

l»liASf:2: ACCREDITATION 

u In 1992, CAEAL will be expanding Its program 
to include not only the provision of test 
samples, but also site visits to observe the 
actual operation of laboratories 
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Protocols are being developed to conduct 
periodic site visits by qualified assessors 



Laboratories, which successfully meet the 
national standards associated with site 
inspections and analyses of test samples, will 
be granted accreditation which will replace the 
certification offered at this time. 
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5.. WHERE ARE WE NOW? 

CAEAL initiated the Certification program early In 
1991 and granted Certification to 40 laboratories 
in Canada. This number is increasing as the 
program develops and more tests are added to 
the list of those offered for certification. 64% of 
participating laboratories are from industry, 19% 
federal and 17% provincial laboratories. 
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All provinces were represented, except PEL 
CAEAL has published a Directory of Certified 
Laboratories. 



El The first of samples were shipped in Feb. 
1991 and the maintenance program calls for 
testing every 6 months 
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For our last study in September 1991, we had 
34% Increase in the number of participating 
laboratories and there are now 55 labs in the 
program 

Samples with organic constituents are being 
developed for distribution in 1992 
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OTHER PROGRAMS 



m EDUCATION 

■r STANDARD REFERENCE MATERIALS 

m RESEARCH AND DEVELOPMENT 

B INTERACTIVE 

n RESOURCE DEVELOPMENT 
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BENEFITS TO MEMBERS 

n Provide a national standard of competence through 
the certification program 

m Provide national recognition for successful 
laboratories through the publication of an annua) 
"Directory of Certified Laboratories" 

m Enhance product quality and reputation of member 
laboratories 
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r A'isist industry to compete in internatiional markets 

p Provide services to members in tlie areas of QA/QC, 
education and training, standard reference materials, 
research and development and communication 

n Al a minimum, tlie establishment of uniform 

performance standards will provide analyses with a 
fixed bench-mark against which to measure their 
performance on a continuous basis 
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The Canadian Association for 
Environmental Analytical Laboratories 

WE'VE MOVED! 



From: Suite 404C - 1 Nicholas 

Ottawa, Ontario 

KIN 7B7 



To our new office 

Suite 532 - 1 Nicholas Street 

Ottawa, Ontario 

KIN 7B7 

Phone: (613) 562-2200 
Fax: (613) 562-2203 
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SECTION FIVE 



QUALITY MANAGEMENT PRESENTATIONS 



CONTROL CHARTING ISSUES - D. KING, MOE 

■VALUATION OF INTER-LAB STUDIES - S. SELLIAH, MOE 



MiSA VVURKSHOP 1992 
CONTROL CHARTING ISSUES 
£>on King 
DISCUSSION 

This presentation shows diagrammaticaUy the rationale for control charting, namely the need to talce timely 
action to limit the relative magnitude of between-day versus within day estimates of precision, to detect 
unusual events (outiiers) and to forestall bias as revealed by drift or discraitinuities. It demonstrates the 
relationships among Shewhait, Youdcn, and King's approaches. It reviews the process for setting control 
limits based on within batch repeatability (Sw) as estimated by the mathematical standard deviation. 

Fi£j_ 

Performance is often discussed in terms of the sharpshooter's bulls-eye. Ability to hit the centre reflects 
accuracy. Spread across the target indicates precision. Since no one is absolutely accurate, accuracy is 
a characteristic of average perfomiance. If the average deviates significantiy from the expected value, the 
perfonmance is biased. Significance is based on the estimate of repeatability (Sw) and the number of 
replicates included in determining the average. The two smaO circles (A, B) represent apparent or actual 
changes in repeatability from day-to-day for a given analyst, or between analysts, or between methods for 
a given analyst. 

When replicate measurements are performed over several batches/runs, or by several analysts/labs, the 
precision is observed to degrade. This reproducibility is attributed to the day-to-day variation in 
restandardizing the measurement system within a single lab (middle large circle), or to the variability of 
standards among labs (bottom large circle). 

In the middle large circle, the small circles represent the variable estimates of repeatability. These are 
centred at the actual bias point for which a standardization adjustment may be required. The line joins the 
individual estimates used to restandardize. The large circle then represents the average reproducibility 
achieved. One aspect of CONTROL is to reduce the size of this circle by controlling daily 
restandardization decisions. 

The lower large circle represents reproducibility amcng laboratories. In addition to the previous effect, 
each lab has a different source of calibration material which must be diluted and prepared for use as a 
daily standard. This induces a source of bias which degrades data comparability among labs, and which 
also must be controlled. 

Fig.2 Youden 2-Sample Plot 

Youden's two-sample procedure, developed in the early '50s for interlaboratory comparison purposes, 
provides evidence for the bias among labs. The result for sample A is plotted versus the result for sample 
B from the same analyst Typically one sees a pattern of points from lower left to upper right. This 
evidence of bias is generally attributed to the differmces among standards in different labs, or to changes 
in method recovery. Results in the other two quadrants are considered very imprecise or erratic. When 
the two samples are similar in concentration these points fisll along a 45 degree axis. The perpendicular 
distance from a point to the 45 degree line is an estimate of that lab's repeatability. The tUstance along 
the 45 degree from the perpendicular to the expectedAargct point represents the lab's bias. The average 






perpendicular distance can be used to determine average repeatability for the participating analysts as 
indicated by the solid circle sunoujiaing tne target pomi. V* ncn li'.c ir-eaiOiT voiuc lor cacn SoiT.pic ocviatcs 
significantly from the target value, the dau can be used to set a consensus point among the labs. An 
analyst deemed as 'biased high' relative to the consensus point may be deemed 'accepubly accurate' 
relative to the target point. 

Fig.3 Shewhan Control Charts 

In the late '20s. Shewhan introduced the lime sequence control chart in which he made replicate 
measurements for a particular characteristic each day, and plowed the nmge (R) and average (X-bar) over 
a period of time. In the lab this requires establishing a control sample which is analysed twice each day. 
The top diagram shows the actual replicate measurements. Many would ignore the apparent trend over 
time. But Shewhan's plot of the average shows the trend quite clearly. 

Performance limits arc determined by using the average range (excluding outliers e.g. > 3 R^^ ) to set 
upper and lower control limits (UCL, LCL) and warning limits (UWL. LWL) for the average. Note that 
the average range can be used to estimate repeatability (Sw = 0.886*R^^g ). The control limit for the 
average is then 3*Ravg*0.886/1.414 or 1.88*Rjyg. The root 2 factor reflects increased confidence in the 
average of two values compared to a single value. 

Fig. 4 Systematic Errors 

In the early '70s King introduced the use of two control samples for controlling daily standardization in 
the lab. One is at the upper end of the operating range: the other is at the lower end. They should be 
close enough together that the repeatability at the two concentration levels is not significantly different. 
Such time-sequence control data can be plotted using the Youden approach. 

When the two samples differ by more than a factor of two (e.g sample A = 25 and sample B = 50 mg/L) 
King observed that there are two '45 degree' lines. One represents a bias which is equal for each sample 
in absolute terms. (Both sample results are low by 5 mg/L for a given analyst.) This type of bias results 
from an error in setting the baseline, background, or blank correction factor. It appears at 45 degrees if 
the X-Y scales are set so that 1 mg/L is a constant distance, leading to a rectangular presentation. 

The other line represents a bias which is coristant in relative terms. (Both sample results are low by 5% 

for a given analyst.) This tvpe of bias represents an error in preparing or using standards, Gack of 
calibration control) or certain types of method failure. This line becomes a 45 degree axis when the X-Y 
scales are set so that 1% is the same distance on both, leading to a square diagram. 

By joining the points in time sequence the systematic drift from day to day is accentuated by the many 
line segments which more or less parallel the bias line from lower left to upper right. 

Fig.5 Two Sample Control 

The data used to plot figiu-e 4 is shown here as two time sequence plots for sample A and sample B. Also 
shown are the Sum plot (A+B) and the difference plot (A-B). In this case the difference represents both 
the repeatability of the test and the slope control status, and the effect of systematic error is essentially 
cancelled on average. (Note that these are signed differences.) Assuming that the repeatabilities at level 
A and level B arc sufficiently similar, then the standani deviation of Ihe differences S^ is approximately 
1.414 larger than Sw as noted by Youden. 
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Control and warning limits for the A-B plot arc set at 3x and 2x Sr,. Outliers rcorcscnt a sudden chanec 

in slope or emtic analysis. The average diffcrcr.::^ indicates siabiii:y or sicpc ana an indication oi Dias 
relative to expected. 

The A+B plot includes a double dose of bias. But when the A-B plot is stable, any larsc variation in the 
A+B plot must be attributed to errors in setting baseline or method blank corrections. On the basis that 
the ratio of two variances is significant when it exceeds a factor of 2, it was decided to allow a factor of 
1.33 times the repeatability as an acceptable level of between-day reproducibility. Therefore the control 
limit for the sums (A+B) is set at 3*1.33*80 or 4*Sj3. The warning limit remains at 2*So. 

Fig.6 Single Sample Control 

Shewhart Control requires at least duplicate analysis of a single control sample. But frequently the analyst 
has only a single analysis per day. Such data can be plotted in time sequence, and the average and 
standard deviation (S) can be detenmined. But such a plot is NOT a control chart in as much as the 
calculated S is used to set limits. This is demonstrated by the top two diagrams. They both contain the 
same data but in different sequence, therefore the average and standard deviation are identical. But there 
is obvious drift in the left hand data set, and there may be a discontinuity shift in the right hand data. 

An estimate of repeatability (Sw) is always required to set proper control limits. This may come from 
other within-run replication data. But it can be estimated from the existing data by examining adjacent 
pairs (Sj(jj). The lower diagrams then are simUar to Shewhart's range chart in calcjiladng an average range 
of adjacent pairs, and using it to set a control limit for the single sample time sequence chart, namely 
3*0.886*Rj,y . One would not add the factor 1.33 used in the King two-sample approach because the 
range of adjacent pairs is probably an overestimate of repeatability. Note that the lower of the two 
estimates (S or S,,jj) is used to set limits. 

SUMMARY 

Bias is readily detected when sufficient replicate measurements are made. The ratio of reproducibility 
over repeatability indicates the level of control over determinate sources of error/bias. Control is directed 
at reducing this ratio on average, forestalling significant drift over time, preventing discontinuities, and 
reacting to outliers as possible evidence for inadequate system operation. Good control is based on two 
principles: 

a) leave well enough alone (don'i fix it if the data is wiLhin control limits), 

b) don't fix it if you don't know what went wrong (have a remedial action plan). 
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A RAPID AUTOMATIC 
GRAPHICAL PROCEDURE FOR 

THE EVALUATION OF 
INTERLABORATORY STUDIES 



SATHI S. SELL! AH 



Historical background: 

SLIDE 1 

In the 50's Youden proposed a scheme where each participant analyses two similar samples. 
They are similar in concentrations. When he received the results he plotted them on a X_Y chart. Each point 
on this graph represents a pair of results from a participant. He divided the graph into four quadrants by two 
lines going through the respective medians. If there are only random errors one would expect the points in the 
graph to be equally distributed among the four quadrants. 

SLIDE 2 

However in actual practice he found that most of labs were in the tower left or upper right quadrants and that 
they tend to lie along the 45 degree diagonal line, if a laboratory reported lower than expected for one sample 
It was very likely that it will report lower than expected for ttie other. Likewise if a laboratory reported higher 
than expected for one sample it is very likely that it will report higher than expected for the other. The systematic 
errors among the laboratories were very evident. 

An acceptance circle based on some factor times the standard deviation is drawn. Laboratories inside the circle 
are deemed satisfactory. Laboratories outside the circle and close to the 45 degree line are biased high or 
biased low. Laboratories outside the circle and lie in the upper left and lower right quadrants are erratic. 

SLIDE 3 

Don King recognized that the Youden plot can be used to isolate two types of systematic errors; those that are 
dependant on the concentration (slope en'ors) and those that are not dependant on the concentration (intercept 
errors). Slope en'ors can be attributed to calibration en-ors (inaccurate standards or inappropriate procedure) 
or recovery problems. Intercept errors can be attributed problems associated with blank's, background or 
baseline corrections. 

In order to monitor these two enx>rs on a day to day basis a two sample control procedure was introduced to 
the MOE laboratories. Two controls A & B which are different in concentrations are analyzed just after the 
calibration standards, each time a new run is set-up. The results of these standards are plotted as shown and 
the points are joined sequentially . The emerging pattem of the AB plot reveals the type of systematic errors 
if there are any. 

In these graphs there are two diagonal lines. The line that goes through the origin represents identifies slope 
biases and the 45 degree line identifies the intercept biases. If ttiere are no biases, the AB pattem will take a 
circular shape. 

SLIDE 4 

The previous two approaches have been combined to evaluate inter-lab data. Here too. each laboratory is 
required to analyze two samples, but the samples are of different concentrations. This primarily enables us to 
distinguish between the two types of systematic errors identified earlier. 

While inter-lab reproducibility S, tends to be concentration dependant, the repeatability S, may not be 
concentration dependant. The dependence of repeatability on concentration is assessed by determining the 
reproducibility for each sample after excluding the outliers and determining the ratio of the S,. 'Fstatistic'is used 
to assess if the difference between the two S;s are different. If S, of the two are not significantly different it is 
concluded that the S, is not concentration dependant. 

When repeatability is independent of tfie concentration the raw data can be used as such (that is in 
concentration unite or absolute scale) to draw the graphs and define the acceptable pertormance criteria. The 



resulting diagram takes a rectangular shape. When repeatability is concentration dependant, then the raw data 
must be transformed into a more comparable scale before acceptable performance cnteria could be determined. 
In such cases the results will be expressed as percentages of target (or median) resulting in a square graph. 
This ensures that the acceptable repeatability zone remains essentially circular in shape. 

In both cases you can identify the two diagonal lines representing the two types of errors. 

SU0E5 

The next step is to calculate the average within-laboratory repeatability S,. This is done by calculating the 
perpendicular distances from each point to the appropriate diagonal line and averaging these PDs. A screening 
process is involved at this step to eliminate highly imprecise or erratic data. Actual steps Involved are 
summarized in slides 7 & 8. 

SUDE6 

The Sw is used to define the warning and control limits. The inner circle represents the limits for repeatability. 

We recognize that additional tolerance is required to allow for the variability in preparing and using calibration 
standards on a day to day or among labs basis. This is the reproducibility component. 

A factor 1 .5 was for ttie ratio of reproducibility to repeatability is considered an achievable goal for interlaboratory 
reproducibility and accordingly the area of acceptable performance is extended in the lower left and upper right 
quadrants have been extended by a factor of 1 .5. 

The evaluation of a laboratory in this procedure will depend on its position on the graph. 

Laboratories that lie within this key hole shaped shaded area will be considered as acceptable performers. 

Laboratories tnat have blank problems will lie close to the intercept bias line. 

Laboratories that have calibration problems will lie along the concentration bias line. 

Laboratories that lie close to any of the median lines have one of their results correct and the other in error. 
They are clearly out of control, 

The laboratories in the upper left and lower right are highly imprecise or erratic. 

SLIDE 9 

A main feature of this method is that it is designed to prompt immediate remedial action when problems are 
identified. 

All Hie steps I have described can be automated. 
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SLIDE 2 
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SLIDE 6 
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SLIDE 7 

Step-wise Summary of The Evaluation Procedure 

1. Sample preparation and distribution 

split two samples of different concentrations among a number of laboratories for 
anaiysis/measurement using their current methodoiogy. 

2. Enter data on LOTUS 123® spreadsheet. 

Caicuiate median (L„,H J, means and standard deviations for each sampie. 
Tabulate data and return to laboratory analyst for verification. 
Correct database If transcriptional errors were reported. 

3. Evaluate high sample data: 

I) reject all results which differ from the median (H„) by more than 10% 

if) calculate medlan(H), mean and standard deviation (SJ 

ill) re-Include data If within 3 times S^ 

iv) reiterate ii) and ill) until no further data Is included 

V) caicuiate relative standard deviation of the final selected data (CV^) 

4. Evaluate low sample data: 

9 use 3 X CVh X medlan(LJ to exclude possible outliers 

Ii) caicuiate medlan(L), mean and standard deviation (S,) 

ill) reinclude data If within 3 times S,) 

Iv) reiterate ii) and lii) until no further data Is Included. 
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SLIDE 8 

5. Determine paired sample performance criteria: 

I) examine the ratio of S^/S, and if: 

<2 use data as reported In concentration units 

otherwise convert to % recovery based on expected value if known 

(otherwise use median values (H,L)) 
il) prepare paired sample scatter diagrams of all data 
ill) calculate perpendicular distances from each point to the two 45 degree 

lines (slope and intercept error lines) and select the lesser of the two 

perpendicular distances (PD) 
iv) determine the median PD 
v) determine the average of all PD values less than 2.5 times the median and use this 

average to estimate the average repeatability S^. 
vi) set warning limits for repeatability = 2 times S^ 
vli) set control limits for repeatability = 3 times S^ 
viil) set warning limits for possible bias = 3 times S^ 
ix) set control limits for possible bias = 4.5 times S^ 

6. Code performance based on location of points on 
the diagram using LOTUS 123® program: 

in upper left or lower right quadrant (erratic) 

in lower left or upper right quadrant (biased low or high) 

on horizontal or vertical axis (out of control) 

on diagonal line through origin (slope or standard problems) 

on diagonal line not through origin (intercept or bianic problems) 

prepare summary table of performance assessment 



SLIDE 9 
KEY FEATURES 

ACTION FOCUSED PROTOCOL 

yiutomated evaluation protocol 

The procedure Is easily amenable to computation 
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